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(57) Abstract 



A 3D graphics generation and display application for PCs and laptops in a business environment includes an authoring mode screen 
which has a list (38) database elements sought to be graphically displayed, along with a list (40) of variables pertaining to the database 
elements. Also, the application presents a menu (42) of graphics attributes, including each of the three spatial dimensions and time, as well 
as various graphics object attributes, such as color, size, rotation, opacity, intensity, texture, etc. The user determines whether the graphics 
objects are balls for a scatter plot, bars for a bar chart, pie-shaped elements for a pie chart, or other 3D representation. By clicking on a 
database variable in the list (40) of variables and dragging it over an attribute on the attribute menu (42), the user can correlate the database 
variable with the graphics attribute. When the user correlates a variable to a graphics attribute, the correlation is noted by a legend (44) that 
is displayed with the menu (42) of attributes. The graphics are displayed in an animated 3D presentation with graphics attributes, including 
their motion over time, determined by the database variables which have been correlated to the attributes. 
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USER-FRIENDLY GRAPHICS GENERATOR 
USING DIRECT MANIPULATION 

This application includes a microfiche appendix of a software listing, the 
copyright to which is retained by the copyright owner. The copyright owner has no 
objection to the reproduction of the patent document or the patent disclosure as it 
appears in the files and records of the U.S. Patent and Trademark Office, but 
otherwise reserves all copyrights whatsoever. 

FIELD OF THE INVENTION 

The present invention relates generally to computer-implemented graphics 
programs, and more particularly to generators for easily making animated, multi- 
dimensional graphics displays for presenting data in an easy to understand visual 
format. 

BACKGROUND 

Effective presentation of computer-stored data is essential for many reasons. 
Among the reasons that effective data presentation is important is that the discovery 
of data trends and correlations between data variables can be promoted by effective 
visual presentation of the data being analyzed. Also, an advocate of a particular 
position might wish to present data in an effective, easy to comprehend format when 
the data supports the position. 

As an example, sales data is routinely stored in databases, and the data can 
represent particular trends and data variable correlations that might be important for 
the selling entity to know, e.g. , sales by product over time, sales by region over time, 
sales by salespeople over time, etc. As another example, computer databases might 
contain information about publicly traded stocks, with the information possibly 
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indicating particular trends and/or correlations between particular stock variables that 
would be important for investors to know. 

From the above discussion, it readily can be appreciated that each tuple or set 
of tuples in, e.g., stock-related data typically represents a database element. Thus, 
a single tuple might represent data on a particular stock on a particular day, with a 
set of tuples representing data on a particular stock over many days. Each tuple 
might include values for many variables. In the stock context, these variables can 
include price, earnings, certain ratios, trading volume, and so on, with a large 
number of potential trend-revealing combinations and correlations being possible 
among the variables. 

Simply presenting the variable values of data tuples in tabular form, however, 
is often a less than effective way to analyze and/or present data. This is because 
people do not quickly discern trends and correlations in data when the data is 
presented in spreadsheet format. Consequently, trends might be present in the data, 
and/or important correlations might exist between variables of the data, but the trends 
and/or correlations might escape discovery when the data is presented in tabular form. 

The present invention recognizes that when data is to be presented to users and 
to business audiences both for analyzing the data and for advocating a particular 
business position, graphical presentation of data can better convey particular business 
trends and forecasts than can simple tabular displays. 

Data tuples may be supplied to computer users in many formats, including, 
among others, real-time data feeds, databases (relational or otherwise), flat files, and 
spreadsheets. The present invention further recognizes that data spreadsheet 
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applications for non-expert computer users have greatly proliferated, and that graphics 
applications that are intended to enable such users to better exploit their databases 
preferably should be relatively simple, indeed, intuitive, to use, so that such users 
might be able to quickly and easily discern correlations and trends as well as generate 
interesting graphics for business presentations. 

Unfortunately, existing personal computer (PC)-based data graphics 
applications merely provide static, two-dimensional graphics charts. To the extent 
that three spatial dimensions are presented by existing PC-based graphics applications, 
the third dimension typically has no data significance. Consequently, such 
applications do not enable the non-expert user to easily depict the potential 
correlations between three or more variables. Further, owing to the generally static 
nature of displays that are generated by existing PC-based graphics applications, time- 
dependent behavior of data is not displayed as effectively as it might be. 

As still further recognized herein, hardware, in the form of computer chips 
to support 3D graphics, has been developed for the computer game industry, and such 
hardware is increasingly being incorporated into PCs for game purposes. The present 
invention propitiously recognizes that such 3D hardware can be used by non-game 
applications, e.g., by a business-oriented application or process-oriented application 
or science-oriented application, to provide a system for generating dynamic 3D 
graphics that can present three or more database variables at once that can move or 
vary over time, to enable non-game users to thereby easily discern data trends and/or 
correlations as well as to generate effective data presentations. 
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Accordingly, it is an object of the present invention to provide a PC-based 3D 
graphics system which graphically presents three or more database variables at once. 
Another object of the present invention is to provide a PC-based 3D graphics 
application which has an open architecture such that user developers of the application 
can alter the application to fit their needs. Still another object of the present 
invention is to provide a PC-based 3D graphics application in which correlations 
between three or more database variables can be graphically presented. Yet another 
object of the present invention is to provide a PC-based 3D graphics application that 
is easy to use and cost effective. Another object of the present invention is to provide 
a PC-based 3D graphics application that presents a dynamic graphics display which 
uses time-dependent variance to enliven the display and to present the values of a 
database variable using the dimension of time. 

SUMMARY OF THE INVENTION 

A computer program product having a computer readable medium is used to 
direct a digital processing apparatus to perform the inventive method steps disclosed 
herein. In other words, the program product is readable by the digital processing 
apparatus, and the program product tangibly embodies a program of instructions 
executable by the digital processing apparatus to perform the inventive steps disclosed 
below. Indeed, a digital processing apparatus itself is disclosed herein whose actions 
are directed by the computer program product to perform specific operational steps 
set forth by the computer program product. As intended by the present invention, the 
program product is realized in a critical machine component that causes the digital 
processing apparatus to perform method steps for generating, for presentation on one 
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or more displays, graphics representative of elements in a database, with each element 
being characterized by one or more variables. 

The program product instructs a computing device to present, on the display, . 
a variable list of at least some of the variables. Also, the program on the program 
product includes presenting, on the display or displays, a graphical list of at least 
some graphics attributes of at least one graphics object. Moreover, the program 
includes permitting a user to correlate at least some of the variables to respective 
graphics attributes, and presenting on the display the graphics object or objects, based 
on the correlation of variables to graphics attributes. The graphics object or objects 
may be displayed with the lists. 

In a preferred embodiment, the permitting step includes permitting a user to 
correlate any variable or collection of variables with one or more graphics attributes, 
one of which attributes is display variance. Other graphics attributes include position 
or extent in spatial dimension, graphics object rotation, graphics object color, 
graphics object size, graphics object visual texture, graphics object visual finish, 
graphics object opacity, and, in a broader sense, scent and tactility. 

Importantly, the program immediately depicts on the display or displays the 
effect on the graphics objects of the correlations of database variables to graphics 
object attributes. Thus, the present program permits direct manipulation of the actual 
graphics display by, in one embodiment, clicking and dragging a database variable 
from the variable list onto a graphics attribute in the graphical list. 

The program also includes presenting a legend of variables together with the 
graphical list in automatic response to the correlation of variables to attributes, as part 
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of the direct manipulation feature of the present invention. Preferably, an object list 
of at least some of the objects is presented in a juxtaposed relationship with the 
graphical list. Thus, the lists and graphics objects are presented on the display 
simultaneously with each other. 

In another aspect, a computer program product having a computer usable 
medium includes logic means for receiving data from a database having data 
elements, with each data element being characterized by one or more values of one 
or more variables. Also, logic means are provided for presenting graphics objects 
on a visual display associated with a computer. As discussed in detail below, each 
graphics object represents a respective one of the data elements, and each graphics 
object is characterized by one or more graphics attributes. Furthermore, logic means 
correlate graphics attributes of at least one of the graphics objects to variables of at 
least one data element using a click and drag operation. 

In still another aspect, a graphics system characterized by an open architecture 
includes a data visualization system. The data visualization system in turn includes 
at least one initial visualization model characterized by predefined visual attributes. 
A visualization model control module is provided which includes logic means for 
permitting a user to define, using a preexisting display model, a graphics scene 
characterized by visual attributes. Also, a visualization model interface permits a 
user to correlate database variables to model attributes. Moreover, a user developer 
can define new display or visualization models, such that a subsequent user of the 
data visualization system can use the initial visualization model or the new 
visualization model. 
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Preferably, the data visualization system includes a visualization scene builder 
for generating 3D objects and their behavior over time in response to the visualization 
model control module. Moreover, in the preferred embodiment the data visualization 
system includes a data parser for receiving input data from ODBC data sources and 
real time data feeds from, e.g., the Internet, and outputting values of database 
variables based thereon, and a rendering engine that transforms a 3D scene to 2D 
format, such as an "AVI" format or a "BMP" format. 

In another aspect, a presentation viewer for presenting on one or more 
displays, graphics representative of elements in a database, each element being 
characterized by one or more variables includes a computer program product having 
computer readable code thereon to enable a computing device to undertake the steps 
of accessing one or more files. The accessed files are files that have been previously 
generated by correlating variables on a variable list of at least some of the variables 
with graphics objects on a graphical list of at least some graphics attributes of at least 
one graphics object. Then, based on the files, the graphics object or objects are 
presented on the display or displays. 

The details of the present invention, both as to its structure and operation, can 
best be understood in reference to the accompanying drawings, in which like 
reference numerals refer to like parts, and in which: 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a perspective view of a digital processing system of the present 
invention; 
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Figure 2 is a schematic diagram of the computer program product of the 
present invention; 

Figure 3 is a flow chart showing the overall graphics generation steps of the 
present invention; 

Figure 4 is a view of the authoring mode screen; 

Figure 5 is a view of the presentation mode screen; 

Figure 6 is a flow chart showing the authoring steps; 

Figure 7 is a flow chart showing optional presentation steps; 

Figure 8 is a view of a bar chart screen showing the drill down mode; 

Figure 9 is a flow chart showing optional display innovation steps for causing 
dynamic display behavior that is database independent; 

Figure 10 is a block diagram of the architecture of the present software- 
implemented graphics generation system; and 

Figure 11 is a block diagram showing 2D/3D data flow in the system. 
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

Referring initially to Figure 1, a system is shown, generally designated 10, for 
generating dynamic, animated graphics displays based on business-related data 
spreadsheets. More particularly, the system 10 generates graphics displays having 
up to three spatial dimensions. Within the display, graphics objects are shown which 
have values for various graphics attributes, including position and extent over each 
of the spatial dimensions, and also including variance over time, i.e., animation. 
Each graphics attribute can represent one or more respective variables in a database. 
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Accordingly, the system 10 presents 3D graphics displays that depict data variables 
in an easy to use and easy to understand format. 

As shown in Figure 1, the system 10 includes a digital data processing 
computer 12 and an associated program storage device 14. In the presently preferred 
embodiment, the computer 12 is an IBM-compatible personal computer or laptop 
computer, although other types of computers, e.g., a computer made by Apple. 
Computer or a type HP 9000/755 computer made by Hewlett-Packard and running 
the Unix-based operating system known as HP-UX 9.05, may be used. 

In accordance with the present invention, the program storage device 14 may 
be implemented by one or more processors within the computer 12 that execute a 
series of computer-executable instructions. These instructions may physically reside, 
for example, in electronic random access memory (RAM) and/or electronic read-only 
memory (ROM) of the computer 12. Alternatively, the instructions may physically 
be contained on a data storage medium, such as a computer diskette 16 shown in 
Figure 2. The machine component is shown in Figure 2 as a combination of program 
code elements A-D in computer readable form that are embodied in a computer-usable 
data medium 18, on the computer diskette 16. Or, such media can also be found in 
semiconductor devices, on magnetic tape, on optical disks, on a DASD array, on 
magnetic tape, on a conventional hard disk drive, or other appropriate data storage 
device. 

Thus, the flow charts and block diagrams disclosed herein illustrate the 
structure of the 3D graphics display application program of the present invention as 
embodied in computer program software. Those skilled in the art will appreciate that 
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the Figures illustrate the structures of computer program code elements that function 
according to this invention. Manifestly, the invention is practiced in one essential 
embodiment by a machine component that renders the computer program code 
elements in a form that instructs a digital processing apparatus (that is, a computer) 
to perform a sequence of operational steps corresponding to those shown in the 
Figures. In an illustrative embodiment of the invention, the computer-executable 
instructions may be lines of compiled C ++ language code, or compiled lines of any 
other appropriate language code such as assembly code, or incorporate other object 
code necessary for the application, e.g., object code used by APIs such as DirectX 
(TM) or Open GL (TM). 

In any case, whether embodied by software on a diskette or other medium or 
embodied by logic circuits resident on a semiconductor chip that is accessible to a 
microprocessor, the present computer usable medium includes logic means for 
undertaking the inventive steps disclosed herein. As yet another alternative, the 
present invention can be implemented by a circuit board (not shown). 

Figure 1 shows that the system 10 can include one or more data input devices, 
such as a keyboard 20 and mouse 22. Other well-known input device configurations 
can be used, e.g., a keypad, trackball, or voice-recognition device. Or, a 3D input 
device can be used. Also, the system 10 includes one or more output devices, such 
as a video monitor 24 and a laser printer 26. It is to be understood that the system 
10 can incorporate other output devices well-known in the art, e.g., graphics printers. 

Now referring to Figure 3, the overall graphics generation steps can be seen. 
The results of these steps are shown in the screen displays represented by Figures 4 
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and 5, and further details of the steps shown in Figure 3 are discussed in relation to 
Figures 6-9 below. While for illustration purposes the computerized steps are shown 
in flow chart format, it is to be understood that because of the return loop to the 
correlation steps, the user's experience might be that the steps are undertaken in 
almost arbitrary order. 

At block 28, a data source is established, either by the user or automatically 
by hard code. For example, in the presently preferred embodiment the system 10 
receives input data for graphical display thereof from a Microsoft Excel™ 
spreadsheet, and more particularly from a data output file of Excel™ in either "CSV" 
or "XLS" formats. Alternatively, the input data can be sourced from a real time data 
feed from, e.g., the Internet. The data source can be accessible via the ODBC 
standard. 

Next, at block 30 the authoring mode screen shown in Figure 4 is presented 
on the monitor 24 (Figure 1). Moving to block 32, the system 10 receives user 
inputs regarding what database variables the user wishes to correlate to what graphics 
attributes. Additionally, the system 10 receives user input regarding what type of 
graphics object is to be displayed, e.g., bars for each database element for 
presentation in bar chart format, or pie segments for a pie chart format, or spheres 
for a scatter plot format, or other graphics object type that might be defined by a 
user. In response to the user inputs, at block 34 the system 10 correlates the 
variables to graphics attributes. Simultaneously with the correlation of variables to 
graphics attributes, at block 36 the system 10 presents a graphical representation of 
the database data in either the authoring mode shown in Figure 4 or the presentation 
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mode shown in Figure 5, along with a graph legend. Aspects of this direct 
manipulation are represented in Figure 3 by flow arrows from block 36 to blocks 32 
and 34. 

The above overall steps can be better appreciated in reference to Figures 4 and 
5. Starting with the authoring mode screen shown in Figure 4, a list 38 of database 
elements received from the database at step 28 in Figure 3 is presented. In the 
example shown, the elements on the list 38 are publicly traded stocks, with each 
element representing, e.g., a respective row or indeed a respective page in the 
spreadsheet. 

Also, a list 40 of database variables is presented with the elements list 38. As 
can be appreciated in reference to Figure 4, the variables on the variables list 40 are 
variables that are related to the elements on the elements list 38. In other words, the 
variables are elements associated with the n* position in a set of database tuples, i.e., 
elements on the elements list 38. The variables on the variables list 40 can represent, 
e.g., columns of the spreadsheet, with the variables being input into the spreadsheet 
manually or from a data source, or with certain variables being determined 
mathematically by the spreadsheet application from other spreadsheet variables in 
accordance with spreadsheet means known in the art. Such mathematically 
determined variables are dependent upon the independent variables from which they 
are determined. For example, two independent variables might be a price of a stock 
and the earnings of the stock, with a ratio of the two (i.e., price/earnings) being 
determined as a dependent variable by the spreadsheet application. 
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Figure 4 also shows that juxtaposed with the lists 38, 40 is a graphics attribute 
list 42. As shown, the graphics attribute list 42 graphically depicts attributes of the 
graphics display to be generated, including positions on the three spatial dimensions 
42x, 42y, 42z that correspond to the three axes in Cartesian coordinates. Also, as 
shown the graphics attributes can include attributes of the objects to be displayed, 
e.g., color, size, intensity, texture, rotation, shinyness, etc. Further, the graphics 
attribute list 42 lists time as an attribute. Indeed, non- visual attributes might be 
listed, e.g., scent of objects in a scent-generating system, or tactility of objects in a 
tactile system. 

With the above disclosure in mind, the preferred means by which a non-expert 
user can easily and intuitively associate a database variable with one or more visual 
attributes can be appreciated. The user manipulates the mouse 22 to position the 
display cursor over a variable on the variable list 40. Then, the user depresses the 
mouse button to lock the variable onto the cursor, and next the user moves the mouse 
as appropriate to drag the variable over a desired graphics attribute on the list 42. 
The user then releases the mouse button, indicating that the variable is to be 
correlated with the graphics attribute. Thus, the user can easily correlate any variable 
with any graphics attribute using a click and drag operation. 

The correlation of a variable with an attribute is automatically indicated, as 
soon as the correlation is made, by a legend 44 that appears next to the graphics 
attribute list 42. In the example shown in Figure 4, a variable "debt to equity" has 
been correlated to the x-axis, and a variable "price-per-share" has been correlated to 
the y-axis. On the other hand, a variable "volume" has been correlated to the z-axis. 
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Additionally, a variable "%drop on day of gap" has been correlated to the graphics 
attribute of object color, whereas a variable "annual sales" has been correlated to the 
graphics attribute of object size. 

In response to and simultaneously with the above correlations, in addition to 
presenting the legend 44, the system 10 presents a graph 46 with up to three spatial 
dimensions (3D) and having, in one preferred embodiment, Cartesian coordinates. 
In other words, the non-expert user does not have to overlay the present graph 46 or 
legend 44 with a dialog box to undertake the correlations or indeed to change grid 
lines, font, etc., and then return to a graph to view the results of the manipulations 
in the dialog box, in contrast to many existing systems. Instead, the graph 46, legend 
44, and necessary dialog elements (i.e., the lists 38, 40, 42 and the below-described 
graphics control elements such as buttons and toolbars) are presented simultaneously 
with each, with the legend 44 and graph 46 changing in response to and 
simultaneously with user inputs. 

Although 3D is preferred for the graph 46, it is to be understood that if the 
user desires to display the values for only one or two variables, the present system 
can render a two-dimensional graphics display. The graph shown in Figure 4 
presents graphics objects 48 in the form of scatter plot spheres, although it is to be 
understood that, as mentioned above, other 3D formats can be presented, e.g., bar 
graphs or pie charts. As shown, the graphics objects 48 on the graph 46 are 
displayed with the lists 38, 40, 42 and the legend 44. 

The positions of the graphics objects 48 in the Cartesian system shown are 
defined by the variables correlated to the respective axes. Accordingly, in the 
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example shown, the x, y, and z coordinates of each object (representing a database 
element on the elements list 38) is respectively determined by that object's debt to 
equity, price per share, and volume. Also, the color of each graphics object 48 is 
determined by the corresponding database element's %drop on day of gap, while the 
size of each graphics object 48 is determined by the annual sales of the corresponding 
database element. If desired, additional database variables on the variables list 40 can 
be correlated to additional graphics attributes, e.g., intensity, texture, rotation, 
shinyness. 

Moreover, the graph shown in Figure 4 can be animated, i.e., the graphics 
attributes of the graphics objects 48 can be caused to vary over time, by correlating 
a database variable to the graphics attribute of time. The present invention can thus 
create animation that can be written to "AVI "-formatted files. To create the 
animation, key frames can be set along the animatable axis at predetermined time 
intervals with the system 10 interpolating between the corresponding camera positions 
to move the graphical presentation from one key frame to the next. Alternatively to 
time, the animatable axis can be any linear progression such as temperature, 
conductivity, or other property. 

As discussed further below, correlation of a database variable with an attribute 
such as variance over time can be done by the user using the click and drag procedure 
described above, or automatically by the system 10. For example, if each database 
element represents a page of a spreadsheet of stock data, with the columns 
representing the variables and the rows representing daily values for the variables, the 
time attribute can be correlated with the row elements corresponding to stock price, 
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so that the graphics objects 48 move over time in accordance to the stock price 
fluctuation of the underlying database element. 

If desired, the authoring mode screen shown in Figure 4 can include various 
graphics control elements such as buttons and toolbars that are operated by means 
known in the art, e.g., by clicking the mouse 22 on the elements or by touching the 
elements if a touch screen display is used. For example, a time window control 
element 50 can be operated by appropriately manipulating the mouse 22 to establish 
the time window over which animation is desired on the graph 46. Also, dimension 
control elements 52, 54, 56, 58 can be used to respectively present the 3D graph 46 
shown in Figure 4, or to present a 2D depiction of the data by displaying an 
orthographic projection of the graph such that the x, or y, or z dimension is viewed 
from the side and, hence, is suppressed. 

Furthermore, a zoom button 60 can be operated to cause a portion of the one 
or more of the lists 38, 40, 42, legend 44, or the graph 46 to be enlarged, and a bide 
button 62 can be operated to remove an object or graph portion from view. Linear 
and spline buttons 64, 66 can be used to select the method for interpolating between 
values of data variables that might be depicted on the graph 46, and a full button 68 
can be operated to invoke the presentation screen shown in Figure 5. If desired, a 
graphics mode button (not shown) can be provided to enable the user to signal to the 
system 10 a desire to convert, e.g., a scatter plot as shown into a bar chart, in which 
the graphics attributes on the graphics attribute list 42 would correspond to attributes 
of the graphics objects (e.g., rectangular bars) of the new graphics mode. 
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Moreover, the user can click on a graphics object 48 and move the object 48. 
In accordance with present principles, when a graphics object 48 is moved or 
otherwise has one of its attributes changed, the system 10 simultaneously causes the 
underlying variable value in the database to be changed appropriately. Thus, the user 
can change the database by manipulating the graphics objects 48. 

Figure 5 shows the presentation screen that the user can invoke, once the 
variable-to-attribute correlations discussed above have been defined. The presentation 
screen shown in Figure 5 is in all essential respects identical to the authoring screen 
shown in Figure 4, with the exception that the elements list 38 and variables list 40 
have been removed. The user can return to the authoring screen by operating an 
author button 70. If desired, just that portion of the system that presents the screen 
shown in Figure 5 can be provided to a user who does not possess the entire system 
10 for enabling such as user to view, but not edit, the graph 46. This presentation 
portion of the system 10 can be configured as a separate application to accept as input 
the relevant variable values and animation controls, so that it can be operated 
independently of the original data source. 

The details of the correlating steps described above can be appreciated in 
reference to Figure 6. At block 72, the system 10 receives data from a database such 
as an Excel™ spreadsheet or other data source, and at block 74 the elements list 38 
(Figure 4) is presented. Next, at block 76 the variables list 40 is presented with the 
elements list. 

In presenting the variables list 40, as indicated at block 76 the variables to be 
presented can be preset or they can be selected by the user. For example, the system 
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10 can infer the variables by determining the column or row headings, as appropriate, 
of a spreadsheet as the variables on the list 40. Or, the user can select variables from 
the data source to be listed. 

Alternatively, the system 10 can infer variables for presentation. For 
example, as indicated above variables can be presented as independent or dependent 
variables, depending upon whether the data source mathematically determines certain 
variables (i.e., dependent variables) as functions of other variables, or whether the 
values of the variables are constant, i.e., are independent variables. If the system 10 
determines that the data source so determines variables, the dependent variables can 
be listed as superordinates to the independent variables from which the dependent 
variables are derived. 

The above consideration is addressed at decision diamond 78, wherein it is 
determined whether the user has indicated that variables are to be presented in 
hierarchical form. If so, the process moves to block 80 to organize the variables on 
the list 40 hierarchically, with the hierarchy being defined by the user or being 
inferred, e.g., from data source algorithms, as described above. At block 81 the data 
can be written to a .pds file described further below. 

From block 80, or from decision diamond 78 if the decision there was 
negative, the process moves to block 81A to permit the user to select a visualization 
model or, if one has already been selected, to change to another model. Thus, at 
block 81 A the user can select to use a bar chart model, pie chart model, scatter plot 
model, or other model supported by the system 10 for presentation. 
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From block 81 A the process moves to block 82, in which the graphics 
attribute list 42 (Figure 4) is presented graphically and alpha-numerically as shown 
in Figure 4. Next, at block 84 the user click and drag commands are received to 
correlate the database variables on the variables list 40 with the graphics attributes on 
the list 42. 

Proceeding to decision diamond 86, it is determined whether automatic time 
correlation has been selected. More specifically, at decision diamond 86 it is 
determined whether the user wishes to correlate a variable with the graphics attribute 
of variance of an attribute (e.g., graphics object position, size, color, etc.) over time, 
or whether such correlation is to be undertaken automatically. If the correlation of 
the time attribute is to be undertaken automatically, the system proceeds to block 88 
to correlate the time attribute to a database variable whose value constitutes a 
sequence by, e.g., inferring that a first data page represents elements and their 
variables at a first time, the next subsequent data page represents the elements and 
variables at a second, subsequent time, and so on, and that the time dependency of 
one or more of the variables is to be shown in animation. Or, the variable that is 
determined to most strongly reflect time can be correlated to the time attribute. 

From block 88, or from decision diamond 86 if the test there is negative, the 
system 10 proceeds to block 90 to normalize the objects 48 on the graphics plot 46 
(Figure 4). In normalizing the data, the system 10 can use conventional 
normalization techniques to normalize the data to the available range of each visual 
attribute. Or, the user can specify a normalization, or the data can be normalized to 
the value of a selected object's variable. As yet another alternative, after initial 
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normalization, one or more graphics objects can be muted as mentioned above, and 
the data then re-normalized, to more effectively plot the remaining data elements. 

Furthermore, at block 90 the display may be time shifted automatically or in 
accordance with a user-defined time shift. Stated differently, the graphics plot 46 can 
shift any visual attribute by selecting a particular value of the corresponding variable 
as a "zero point" for the scale of the attribute for each element in the 
database/spreadsheet. For example, using the stock analogy, it may be desirable to 
define the zero point for each one of a number of stocks not according to an absolute 
time standard, but according to when a particular event, e.g., a so-called "gap" day, 
occurred for each stock. It will be appreciated that under such circumstances, the 
database variables that have been correlated to the graphics attribute of time are 
displayed in the temporal dimension according to a reference which is related to the 
elements, and not to an absolute time reference. 

At block 92, the graphics objects are displayed with the appropriate graphics 
attributes as described above. To represent the direct manipulation of the present 
invention, Figure 6 shows double flow arrows from block 92 to and from states "A", 
"B", and "C", respectively representing blocks 76, 84, and 81 A. In other words, 
Figure 6 reiterates that as a user correlates a database variable to a graphics attribute 
by the above-described click and drag process, the graphical display and legend 44 
are simultaneously updated as appropriate to reflect the correlation, and Figure 6 
further reiterates that the user may change on the fly from one visualization model 
(e.g., bar chart format) to another visualization model (e.g., pie chart format). 
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Now referring to Figure 7, further details of the presentation mode can be 
seen. Commencing at decision diamond 94, it is determined whether the user has 
selected a "tear off" section of the spreadsheet for presentation by, e.g., selecting a 
portion of an Excel™ spreadsheet. If so, the process moves to block 96 to 
graphically display the "tear off" after variable to attribute correlation, normalization, 
and time synchronization as discussed above. 

Proceeding to decision diamond 98, the system determines whether the user 
has selected a new look for the graphical representation. If so, the process moves to 
block 100 to change the look of the graph 46 (Figure 4) by changing, as determined 
by the user, any one or more of the grid style, grid color, lighting model, font, object 
configuration (e.g., by displaying points in a scatter plot as fanciful objects instead 
of spheres), and object finish. 

The next user interaction supported by the present system is shown starting at 
decision diamond 102, wherein it is determined whether the user has specified that 
the user desires to have a key time period of data or other key sequence of data saved 
for future display. If the user has designated a key sequence by appropriate 
manipulation of the mouse 22, the process moves to block 104 to save the sequence. 

Continuing with the description of Figure 7, at decision diamond 106 it is 
determined whether the user has selected a hierarchical graphical display of data, also 
referred to as a "drill down" presentation. As discussed above, the user can specify 
a hierarchical data organization, and if the user has indicated the desire to graphically 
display the data hierarchically by double clicking on the graphical representation of 
the consolidation (dependent) variable, the process moves to block 108 to replicate 
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the current scene once for each subordinate variable. If desired, the subordinate 
variable graphical representations can be overlaid on the consolidation variable 
representation, or the subordinate variable representations can replace the 
consolidation variable representation. Then, the subordinate variables are time 
synchronized at block 110 using the time synchronization principles discussed 
previously. 

The results of the above process can be appreciated in the bar chart depicted 
in Figure 8, which shows the database variables listed on the left side of the screen 
display and a bar chart on the right. The bar chart shows the accounts receivable 
consolidation variable broken down by region as indicated by the subordinate variable 
bar representations, which are color coded by region. Moreover, the user has 
highlighted the "northeast" subordinate variable on the variables list, causing the bars 
relating to northeast accounts receivable to be exploded away from the bars 
representing the other regions. 

Next, at block 112 user commands are received to vary display characteristics, 
including graphics object 48 opacity, graphics object field depth, etc. Again using 
the stock analogy as an example, the user can specify that unprofitable companies are 
less opaque than profitable companies. To reflect the above-described direct 
manipulation feature, flow arrows extend from blocks 96 and 112 to states "A" and 
"B" . If desired, the output of block 1 12 can be written to a .pds file, details of which 
are disclosed below, at block 113. 

Now referring to Figure 9, various display innovations of the present invention 
are shown. These innovations are essentially trigger mechanisms that cause certain 
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graphics behaviors based on a triggering event, preferred examples of which are 
discussed below. 

At decision diamond 114 it is determined whether the user has selected any 
innovations, and if not, the process ends. Otherwise, the process moves to block 116 
to undertake the novel user-defined, non-database dependent innovations shown in 
blocks 118-122. More specifically, the present display innovations result in animation 
that is not directly dependent on database input, but only indirectly dependent on the 
database. For example, at block 1 18 the user can specify that one object can fade out 
when it approaches another object to within a predetermined distance. Or, as 
indicated at block 120 the user can specify that an object oscillate when the object 
moves below, e.g., the x-axis. As another example of the present display 
innovations, at block 122 the user can specify that an object split into its sub- 
components, if any, when the size of the object exceeds a predetermined size. Using 
the scatter chart stock context shown in Figures 4 and 5 as an example, if the sales 
of a stock (and, hence, the size of its corresponding graphics object 48) exceeds a 
certain point, the graphics object 48 can be split into three smaller spheres, with one 
sphere representing sales in North America, another representing sales in Europe, and 
the third representing sales in the Orient. 

Still further, the present trigger mechanism could pause an animation or switch 
visualization models on the fly, in response to a triggering event in the underlying 
database or in the graphical display itself. The trigger mechanism could thus pinpoint 
an anomalous variable value by pausing the animation when the value is presented, 
to thereby highlight the value. 
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Figures 10 and 11 illustrate the open architecture of the present system 10, 
which allows end users of the present visualization system to generate graphics scenes 
using existing display models (e.g., bar charts, pie charts, scatter plots) and which 
allows user-developers to add new display models to the system in accordance with 
their own preferences. In other words, a non-developer user can use the system 
described above in relation to Figures 1-9, the architecture of which is shown in 
Figures 10 and 1 1, to easily generate effective graphical presentation of data, whereas 
a user developer can use the open architecture shown in Figures 10 and 11 to further 
define display models that can be used by the system 10. 

Data files 124, 126 from a data source are input to a data parser 128. More 
specifically, data in a spreadsheet file 124 (denoted M .xls" to indicate an Excel™ file) 
can be input into the system 10. Also, data from an appropriate data source using 
an appropriate object linking and embedding (OLE) interface to the source that is 
being executed on the computer 12 can be used as a data source. Among the 
applications that contain OLE interfaces are Excel™ and Access™. In the case of 
Excel™, a plug in button can be added to its toolbar to allow a user to directly import 
selected spreadsheet data into the system 10. Real time data input, e.g., from a 
scientific instrument, can also be used. 

The input data is parsed by a data parser 128 into the database variables. In 
one preferred embodiment, the data parser 128 uses a Smart Wizard™ data input 
mechanism. The output of the parser 128 is in the form of data streams 130, which 
can be converted to output files 132, 134 respectively having spreadsheet formats and 
OLE formats, or other formats. 
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Additionally, the data streams .130 are sent to a visualization model control 
module 136. As shown in Figure 10, the visualization model control module 136 also 
receives inputs from a visualization model interface 138 and a standard user interface 
140, which are implemented as part of an application window 142. In one 
embodiment, the application window 142 can be an OLE/ ActiveX™ container. In 
accordance with the present invention, the visualization model control module 136 
includes logic for permitting an end user to define a scene using one or more of the 
visualization display models existing in the system 10 as described above, and to 
characterize the model by visual attributes using the visualization model interface 138. 

If desired, the visualization model control module 136 can output a data 
stream file 144 (designated herein " .pds"). The output file 144 contains both the data 
values being represented, and its graphical correlations. The output file 144 can be 
electronically stored for subsequent reference. 

Also, a visualization scene builder 146 receives input from the visualization 
model control module 136 for generating 3D graphical objects in response to the 
graphics definitions undertaken by the visualization model control module 136. The 
output of the scene builder 146 are 3D objects 148. 

In addition, Figure 10 shows that the open architecture of the present invention 
includes a user-developer application model 143, the purpose of which is to permit 
a user developer to specify new display models by augmenting the visualization model 
interface 138 and the standard user interface 140 to cause the visualization model 
control module 136 to produce the new display models. For example, a user 
developer might want to specify a bar chart model that has not been implemented in 
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the system 10 purchased by the user developer. The user developer uses the 
interfaces 138, 140 to define in both alpha-numeric terms and in graphics the 
graphical shape and graphical attributes of the desired new model. With the above- 
described open architecture, the data visualization system 10 can include initial 
visualization models, each being characterized by predefined visual attributes, and a 
user developer of the system 10 can define a new model and then use the initial 
visualization models or the new visualization model. 

Figure 11 shows the architecture of the system 10 for transforming the 3D 
objects output by the architecture shown in Figure 10 into 2D representations that can 
be presented on a flat display, such as the monitor 24 shown in Figure 1. In general, 
a rendering engine 150 receives data structures in the form of 3D object information 
152 as input and sends the objects 152 to a 2D interface 154. The 2D interface 154 
also receives both animation ("AVI") and still graphics ("BMP") files 155 that can 
be modified by a user to modify flat surface images, e.g., to superimpose a logo on 
a 2D object such as a disc. 

The 2D interface 154 combines the various inputs to present graphics that can 
be presented on a 2D monitor 24. In addition to driving the monitor 24, the output 
of the 2D interface 154 can be output in the form of AVI files 158, BMP files 160 
(if no animation is to be used), or to the printer 26. 

Figure 11 shows that the 3D objects 152 can include mesh objects 162, i.e., 
objects such as bars, spheres, etc. that have volume. Also, in accordance with 3D 
graphics principles known in the art the 3D objects can include camera objects (i.e., 
points of apparent viewing perspective having 3D coordinates), lighting objects 166, 
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and other 3D objects. The lighting objects 166 are points in space from which light 
is modelled to emanate, for, e.g., properly highlighting the graphics objects 48 shown 
in Figures 4 and 5. 

With particular attention to the rendering engine 150, it is to be understood 
that the rendering engine 150 converts the 3D objects 152 into 2D objects that appear 
to have three spatial dimensions in accordance with 3D graphics means known in the 
art. To undertake this conversion, the rendering engine 150 includes subset 
application programming interfaces (API) 168, 170, 172, such as the freely 
distributed APIs known by the tradenames OpenGL, Direct3D, and VRML. 

While the particular USER-FRIENDLY GRAPHICS GENERATOR USING 
DIRECT MANIPULATION as herein shown and described in detail is fully capable 
of attaining the above-described objects of the invention, it is to be understood that 
it is the presently preferred embodiment of the present invention and is thus 
representative of the subject matter which is broadly contemplated by the present 
invention, that the scope of the present invention fully encompasses other 
embodiments which may become obvious to those skilled in the art, and that the 
scope of the present invention is accordingly to be limited by nothing other than the 
appended claims, in which reference to an element in the singular is not intended to 
mean "one and only one" unless explicitly so stated, but rather "one or more". 
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WHAT IS CLAIMED IS: 

1. A computer-based system for generating, for presentation on one or 
more displays, graphics representative of elements in a database, each element being 
characterized by one or more variables, comprising: 

a computer program product having computer readable code thereon 
to enable a computing device to undertake the steps of: 

presenting, on the display, a variable list of at least some of the 
variables; 

presenting, on the display, a graphical list of at least some 
graphics attributes of at least one graphics object; 

permitting a user to correlate at least some of the variables to 
respective graphics attributes; and 

presenting on the display the graphics object or objects, based 
on the correlation of variables to graphics attributes, the graphics 
object being displayed with the lists. 

2. The system of Claim 1 , wherein the permitting step includes permitting 
a user to correlate any variable with any graphics attribute. 

3. The system of Claim 1, wherein the computer program product is 
programmed to further undertake the step of presenting a legend of variables with the 
graphical list in response to an output of the permitting step. 

4. The system of Claim 1, wherein the computer program product is 
programmed to further undertake the step of presenting an element list of at least 
some of the elements, the element list being presented with the graphical list. 
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5. The system of Claim 1, wherein one of the graphics attributes is any 
sequence over time. 

6. The system of Claim 5, wherein the sequence is a variable having time 

values. 

7. The system of Claim 1, wherein the lists and graphics objects are 
presented on the display simultaneously with each other, and wherein the graphics 
objects are manipulated directly by means of the permitting step. 

8. The system of Claim 1, wherein the graphics object is characterized 
by a shape, and the computer program product is programmed to further undertake 
the step of permitting a user to determine at least one of: the shape of the graphics 
object, and the graphics attributes. 

9. The system of Claim 1, wherein the graphics attributes are selected 
from the group of attributes including: spatial dimension, time, graphics object 
rotation, graphics object color, graphics object size, graphics object opacity, graphics 
object visual texture, graphics object visual finish, scent, and tactility. 

10. A computer program product including a computer usable medium, the 
computer usable medium comprising: 

logic means for receiving data from a database having data elements, 
each data element being characterized by one or more variables; 

logic means for presenting graphics objects on a visual display 
associated with a computer, each graphics object representing a respective one 
of the data elements, each graphics object being characterized by at least one 
graphics attribute; and 
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logic means for correlating at least one graphics attribute of at least 
one of the graphics objects to at least one variable of at least one data element 
using a click and drag operation, thereby directly manipulating the graphics 
object. 

11. The computer program product of Claim 10, wherein the means for 
correlating permits any variable to be correlated to any graphics attribute, including 
a variance over time attribute. 

12. The computer program product of Claim 10, wherein each graphics 
object is characterized by at least two attributes respectively representing two 
variables. 

13. The computer program product of Claim 10, wherein each graphics 
object is characterized by at least three attributes respectively representing three 
variables. 

14. The computer program product of Claim 10, further comprising: 
logic means for presenting, on the display, a variable list of at least 

some of the variables; and 

logic means for presenting, on the display, a graphical list of at least 
some graphics attributes of at least one graphics object, wherein the graphics 
objects are displayed with the lists. 

15. The computer program product of Claim 14, further comprising logic 
means for presenting a legend of variables with the graphical list. 
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16. The computer program product of Claim 10, further comprising logic 
means for presenting an element list of at least some of the elements, the element list 
being presented with the graphical list. 

17. The computer program product of Claim 10, wherein the graphics 
objects are characterized by shapes, and the computer program product further 
includes logic means for permitting a user to determine at least one of: the shapes of 
the graphics objects, and the graphics attributes, the graphics attributes being selected 
from the group of attributes including: spatial dimension, time, graphics object color, 
graphics object rotation, graphics object size, graphics object visual texture, graphics 
object visual finish, graphics object opacity, scent, and tactility. 

18. A graphics system having an open architecture comprising: 

a data visualization system including at least one initial visualization 
model, the initial model being characterized by predefined visual attributes; 

a visualization model control module including logic means for 
permitting a user to define a new visualization model and to characterize the 
model by visual attributes; and 

a visualization model interface for permitting a user to correlate 
database variables to model attributes, such that a user of the data 
visualization system can use the initial visualization model or the new 
visualization model. 

19. The data visualization system of Claim 18, further comprising a 
visualization scene builder for generating 3D objects in response to the visualization 
model control module. 
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20. The data visualization system of Claim 19, further comprising: 

a data parser for receiving input data and outputting database variables 
based thereon; and 

a rendering engine for transforming a 3D data object to 2D format. 

21 . A presentation viewer for presenting on one or more displays, graphics 
representative of elements in a database, each element being characterized by one or 
more variables, comprising: 

a computer program product having computer readable code thereon 
to enable a computing device to undertake the steps of: 

accessing one or more files, the one or more files having been 
generated by correlating variables a variable list of at least some of the 
variables with graphics objects on a graphical list of at least some 
graphics attributes of at least one graphics object; and 

presenting on the display the graphics object, based on the files. 



WO 99/27495 PCT/US98/00616 



1/9 




16 







3 





32 



FIG. 2 



Establish 
data source 



28 



I_> 



30 



Present Authoring 
Mode screen 
on monitor 



Receive 
user inputs 
re correlation 
graphics object 
type (bar, pie 
scatter, etc) 




Present Presentation 
Mode on monitor 
w/ legend 



Correlate variables 
to graphics 
attributes per 
user input 



FIG. 3 



•36 



•34 



SUBSTITUTE SHEET (RULE 26) 



WO 99/27495 



PCT/US98/00616 




SUBSTITUTE SHEET (RULE 26) 



WO 99/27495 



PCT/US98/00616 




SUBSTITUTE SHEET (RULE 26) 



WO 99/27495 



PCT/US98/00616 



72 



Receive data 
from db 



FIG. 6 



4/9 



/74 


list variables 
per preset, or user 

selection, or 

inteference 


Present 
objects list 









76 




Organize variables 
hierarchically in 
variable list per user 
selection /inference 





♦ 


NO 


Select/change 




vis. model 



80 



Write 
.pds file 

2 



81 A 



84- 



Receive user click 
& drag commands 

to correlate variables 
to attributes 




Present graphics 
attribute list 
graphically and 
alpha-numerically 





82 



88 ^ 




Correlate time 
dependent variable 

(by inferring data 
page, or largest time 
change) to time att. 




Display graphics 
objects w/ 
graphics attributes 



*92 



Normalize and 

time shift 
automatically or 
per user selection 



90 



SUBSTITUTE SHEET (RULE 26) 



WO 99/27495 



PCT/US98/00616 



5/9 



No 




96 



Display tearoff 
after normalization, 
time sync. 




Change look (grid 
style, color, lighting 
model, font, object 
configuration/finish etc.) 



FIG. 7 



<5> 



Receive user 
commands to 
vary object opacity 
field depth, etc. 



Time Synchronize 
subordinate variables 





110 



108 



Replicate current 
scene for each 
subordinate variable; 
overlay on consolidation 
var. or replace 



piinc-rrn rrc cucct /ni n c oc\ 



WO 99/27495 



PCT/US98/00616 



6/9 




SUBSTITUTE SHEET (RULE 26) 



WO 99/27495 



PCT/US98/00616 



7/9 




Fade one object os 
it approaches other 
at predetermined 
distance 



118 



Cause object 
to oscillate when 

object below 
predetermined axis 



120 



Split object 
into sub-component 

when object size 
predetermined size 



122 



SUBSTITUTE SHEET (RULE 26) 



WO 99/27495 



PCT/US98/00616 



-124 



y 1 


/ 




/ 






*.xls 






OLE 















-126 



8/9 
FIG. 10 



-C- 



142 



Application Window 
OLE/ActiveX container 



128 



Data Parser 



(Smart Wizard 
Data Input) 



143 



138 



User-Developer 
app. module 



130 



Data Streams 



Visualization 

Model Interfaces (Ul) 

(Bar, line, pie, ...etc) 



136 



visualization 
Model Controls 

(Bar, line, pie, ...etc) 



-134 





/ 


/ 


/ 




/ —r 


*.xls 




OLE 






*.pds 









144 



132 



-C- 



140 



Standard Ul 
Menus 

Generic office 
toolbar 



146 



Visualizer Scene 
Builder 




SUBSTITUTE SHEET (RULE 26) 



WO 99/27495 



PCT/US98/00616 



■155 





/ 




/ * 


AVI 






BMP 







9/9 
FIG. 11 




SUBSTITUTE SHEET (RULE 26) 



INTERNATIONAL SEARCH REPORT 



International application No. 
PCT/US98/00616 



A. CLASSIFICATION OF SUBJECT MATTER 
IPC(6) :G06T 1 1/00 
US CL .345/440, 326 

According to International Patent Classification (IPC) or to both national classification and IPC 



B. FIELDS SEARCHED 



Minimum documentation searched (classification system followed by classification symbols) 
U.S. : 345/440,326 



Documentation searched other than minimum documentation to the extent that such documents are included in the fields searched 



Electronic data base consulted during the international search (name of data base and, where practicable, search terms used) 



DOCUMENTS CONSIDERED TO BE RELEVANT 



Category* 



Citation of document, with indication, where appropriate, of the relevant passages 



Relevant to claim No. 



US 5,657,460 A (EGAN ET AL.) 12 AUGUST 1997, figs 4a-4c, 
figs 8-10, fig 16, columns 10-12. 



1-8, 10-16, 18, 
19, 21. 



|~] Further documents are listed in the continuation of Box C. See patent family annex. 




t pubUwl after the i 



das* and not m con f lict with the 



ad filing daas or priority 



document of pcrtfcufar re Wv once; the claimed swanm 
considered novel or canoe* be ww iw icfcd to invoHs «n i 
when the document is taken done 



eat of particular re l e v ance; the claimed ai petition 
to involve an inventive step when tfc 



demefttased 



am, exhibition or other 
filing date but later than 



being obvious to a person skilled in the an 
document member of the same patent family 



Data of the actual completion of the international search 
21 APRIL 1998 



Date of mailing of the international search report 



I 0 AUG \m 




Name and mailing address of the ISAAJS 
Coeannasieoaf of relents and Trademarks 
BoxPCT ':. 
Wastages, IXC 20231 

Facsimile No, C7O3)305-323O 



Aphorized officer 

^LMIS JANKUS 
Telephone No. (703) 305j 



